Single-Lens 3-D Imaging Device Using Polarization Coded Aperture Masks Combined With Polarization Sensitive Sensor

ABSTRACT

A device and method for three-dimensional (3-D) imaging using a defocusing technique is disclosed. The device comprises a lens, at least one polarization-coded aperture obstructing the lens, a polarization-sensitive sensor operable for capturing electromagnetic radiation transmitted from an object through the lens and the at least one polarization-coded aperture, and a processor communicatively connected with the sensor for processing the sensor information and producing a 3-D image of the object.

CROSS-RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/007,765, filed Jan. 17, 2011, which is a continuation of U.S. patentapplication Ser. No. 12/150,238 filed Apr. 23, 2008, which claims thebenefit of priority of U.S. Provisional Application Ser. No. 60/925,918,filed Apr. 23, 2007, U.S. Provisional Application Ser. No. 60/926,010,filed Apr. 23, 2007, and U.S. Provisional Application Ser. No.60/926,023, filed Apr. 23, 2007, each of which is fully incorporatedherein by reference.

BACKGROUND OF THE INVENTION

(1) Technical Field

The present invention is related to a device and method forthree-dimensional (3-D) imaging and, more specifically, to a single-lens3-D imaging device using a polarization-coded aperture mask combinedwith a polarization-sensitive sensor.

(2) Background

Three-dimensional (3-D) imaging is a continuously evolving field thatwould benefit from improved imaging techniques. Enhanced 3-D imagingcould be used for a variety of purposes, such as to generatequantitative information about an imaged object (through quantitative3-D imaging). However, existing imaging techniques have failed tosufficiently support quantitative 3-D imagining. For example, when apoint that is not on the focal plane of an imaging system is imagedthrough the imaging system, the captured point detected by a sensor issaid to be defocused. If the imaging system has a large aperture, thenthe defocused point will appear blurred. For this reason, it has beensuggested that the blur of the image of a point can be used toquantitatively determine the distance from that point to the focal planein space. It has also been suggested that if the position of the focalplane is known, the imaging system could be used for quantitative 3-Dimaging. To reconstruct the 3-D position of a point, it is onlynecessary to measure the size and/or intensity of the blur disc (Z) andthe point position on the sensor (X, Y).

In practice, however, such a system is difficult to effectivelyimplement. First, a blurred image occupies a large amount of space onthe sensor, so sophisticated algorithms to separate overlapped imagesare necessary. Second, the amount of light entering the optical systemdoes not change appreciably between a focused point and a defocusedpoint (unless the focal plane is very close to the optical system).Thus, the blurred image puts the same amount of energy onto the sensoras a focused image, but spread over a larger area. The intensity of adefocused image is inversely proportional to its area, so a quantitativemeasurement of the distance between the focal plane and a point basedonly on blur requires a sensor with an extremely high dynamic range. Inreal lenses, there are also diffraction effects which make blurredimages look more like rings than broad Gaussian distributions in certaindepth ranges, making software processing complicated. See, for example,Wu, M.; Roberts, J. W.; and Buckley, M., “Three-dimensional fluorescentparticle tracking at micron-scale using a single camera,” Experiments inFluids, 2005, 38, 461-465. Even without lens aberrations or diffraction,image processing is complicated by the fact that since the depthinformation comes from a measure of the diameter of a blur spot, theintensity of the imaged point affects the measurement. For example, iftwo defocused points A and B have the same amount of defocus, but pointA is brighter than point B, typically point B's image will be measuredas having a smaller diameter than point A's simply because it does notrise as far from the background illumination in the scene.

The original “defocusing” concept recognized that in such a blur-basedsystem, the depth information is carried only by the marginal (outer)rays of the ray pencil that forms the image. See, for example, Willert,C. E.; and Gharib, M., “Three-dimensional particle imaging with a singlecamera,” Experiments in Fluids, 1992, 12, 353-358. It is the angle thatthese rays make with the sensor plane that dictates the sensitivity ofthe imaging system. Thus, an equivalent measurement should be possibleby placing small apertures off-axis in the imaging system, such thatonly marginal rays may pass through to form an image. If a blur system,as described above, has its large aperture replaced with a smallaperture placed anywhere on the circumference of the large aperture,then the image of a defocused point is now a small spot located on whatwould otherwise be the circumference of a blurred image. The end resultis depth information that is transmitted not by the size of a blurredspot, but rather by a lateral offset in a much smaller spot. Measuringthe location of a spot on an image is much less sensitive to intensitydifferences than measuring its size.

The use of small apertures alleviates the dynamic range issues with ablur-based system, since the high f-number of the small aperture makesdiffraction blur (not defocus blur) the primary blurring agent in theimage. This means that within a large range of distances from the focalplane, the images are almost the same size.

Using off-axis apertures means that reconstruction of a point's positionin space now involves finding all the images of a single point on thesensor and measuring the distance between them. The images will appearin the same pattern as the aperture arrangement; for example, if threesmall apertures arranged as vertices of an equilateral triangle areused, then the image of a defocused point is three small spots arrangedin an equilateral triangle. The orientation of the images' trianglerelative to the apertures' triangle reveals whether the defocused pointis ahead of or in front of the focal plane. Additionally, the size ofthe images' triangle relates to the distance between the defocused pointand the focal plane. The size of the triangle is zero for a focusedpoint which occurs when all three images are on top of each other. Thesize of the triangle increases as the amount of defocus increases.Multiple small images take up less space on the sensor than one largeblurred one, so the overlap problem is alleviated by this arrangement.

The matching problem in the reconstruction creates a new problem; if theobject being imaged is a set of featureless points, then the images areindistinguishable and can only be matched according to their relativelocation (for example, finding all dots on an image that formequilateral triangles within some tolerance). This relatively loosematching criterion necessitates that three or more apertures be used toreduce the number of mismatches or “ghosts.”

A single off-axis aperture records depth information; however, Z cannotbe separated from the in-plane position of the point imaged. Twoapertures record the depth information and allow the in-plane positionto be extracted independently of Z. In practice, it is impossible toreconstruct a random point cloud with only two apertures because manyghost particles are generated when images are mismatched. Moreover, itis impossible to know if a particle was in front of or behind the focalplane from only two images. With three apertures, mismatches are reducedand the sign of the distance from the particle to the focal plane isknown by the orientation of the triangle formed by the images.

See, for example, Willert, C. E.; and Gharib, M., “Three-dimensionalparticle imaging with a single camera,” Experiments in Fluids, 1992, 12,353-358.

The original practical implementation of the defocusing concept consistsof a single lens with three off-axis apertures imaging onto a singlemonochromatic sensor (i.e., three was deemed the minimum number ofapertures that produced acceptable results). It should be noted thatbecause the defocusing measurement is a measurement of a point'sposition relative to the focal plane, it is necessary to know theposition of the device to know the absolute position of desired point.

The three off-axis apertures imaging onto a single monochromatic sensoralso has disadvantages. Overcrowding of the sensor is still an issuewhen the point density within the scene is high. In this case, eachpoint has up to three images on the sensor and there is still a possibledynamic range issue (i.e., a point on the focal plane will have threeimages that coincide on the sensor and thus will look three times asbright as defocused points). The dynamic range issue can be overcome byselectively illuminating the volume so that no points on the focal planeare imaged.

As described in U.S. Pat. Nos. 6,955,656 and 7,006,132, one solution tothe overcrowding problem is to image each aperture with a separatesensor. This adds to the matching criterion, because now each spot onthe image can only be one of the vertices of the aperture arrangement;since the source (aperture) of each spot is known, there is slightlyless ambiguity in the matching process.

Further, the addition of more sensors (for example, a charge-coupleddevice (CCD)) has the disadvantages of higher cost and larger size(along with manufacturing complications) relative to a single-sensorsystem. Moreover, multiple-sensor arrangements pose alignment challengesand robustness challenges; the multiple sensors are also differentlyaffected by temperature, vibration, and other environmental effects andas such are more prone to calibration errors.

For the foregoing reasons, there is a need for a quantitative 3-Dimaging system which either alleviates or eliminates the matchingproblem. The system should be viable in a single-lens, single-sensorarrangement for simplicity and compactness and also should be easilyexpandable to a multiple-lens, multiple-sensor arrangement if sodesired.

SUMMARY OF THE INVENTION

The present invention is related to a device and method forthree-dimensional (3-D) imaging and, more specifically, to a single-lens3-D imaging device using a polarization-coded aperture mask combinedwith a polarization-sensitive sensor.

The device comprises a lens, at least one polarization-coded apertureobstructing the lens, a polarization-sensitive sensor operable forcapturing electromagnetic radiation transmitted from an object throughthe lens and the at least one polarization-coded aperture, and aprocessor communicatively connected with the sensor for processing thesensor information and producing a 3-D image of the object.

In another embodiment, the mask has two mutually distinctpolarization-coded apertures, a relative polarization of the twoapertures is approximately perpendicular, and the polarization-sensitivesensor is configured to distinguish between images generated by eachaperture, thereby minimizing mismatching.

In yet another embodiment of the present invention, the at least onepolarization-coded aperture rotates such that the polarization of theaperture changes between exposures, whereby depth information can becalculated from a distance between images of the same marker ondifferent exposures.

As can be appreciated by one skilled in the art, the present inventionalso comprises a corresponding method of 3-D imaging, the methodcomprising acts of capturing electromagnetic radiation transmitted offof an object and through a lens and at least one polarization-codedaperture with a polarization-sensitive sensor, and processinginformation from the sensor to produce a 3-D image representative of theobject.

In another embodiment of the method of the present invention, the maskhas two mutually distinct polarization-coded apertures, a relativepolarization of the two apertures is approximately perpendicular, andthe polarization-sensitive sensor is configured to distinguish betweenimages generated by each aperture, thereby minimizing mismatching.

In yet another embodiment, the method further comprises an act ofrotating the at least one polarization-coded aperture such that thepolarization of the aperture changes between exposures, whereby depthinformation can be calculated from a distance between images of the samemarker on different exposures.

BRIEF DESCRIPTION OF THE DRAWINGS

The objects, features and advantages of the present invention will beapparent from the following detailed descriptions of the disclosedaspects of the invention in conjunction with reference to the followingdrawings, where:

FIG. 1A is an illustration showing a band-pass filter system thatincludes a sensor;

FIG. 1B is an illustration showing a defocused, multiple, pattern-codedimage acquisition of real points as received by the sensor of FIG. 1A;

FIG. 1C is an enhanced-view illustration showing the framed area of FIG.1B, demonstrating the matching procedure for a multi-wavelengthaddressable-pattern in the form of a red dot and its corresponding greendot;

FIG. 1D is a illustration showing a chart of the relationship of focallength (L) to Z-distance of matches and “ghost” particles with respectto FIG. 1C;

FIG. 2 is an illustration showing a polarized filter imaging system;

FIG. 3 is an illustration showing an aperture system for imaging points;

FIG. 4A is an illustration showing a synched, single-aperture systemwith a single-hole mask shown in a first position A;

FIG. 4B is an illustration showing a synched single-aperture system witha single-hole mask shown in a second position B;

FIG. 4C is an illustration showing a rotatable aperture along with theimages of two objects produced at different angles of rotation;

FIG. 5A is an illustration showing a single-aperture system havingmultiple f-stops;

FIG. 5B is an illustration showing an image acquired from the sensor ofa single-aperture, multiple f-stop system;

FIG. 5C is an enhanced-view illustration showing the framed area of FIG.5B;

FIG. 5D is an illustration showing a chart of matched points asdetermined by a processor;

FIG. 5E is an illustration showing a vibrating, single-aperture system;

FIG. 5F is an illustration showing an asymmetrical aperture, and acomparison view of the corresponding images produced by an object infront of versus in back of the focal plane;

FIG. 5G is an illustration showing an embodiment with a large centralaperture;

FIG. 5H is an illustration showing an aperture with a light projector;

FIG. 6A is an illustration showing an electronically masked imagingsystem with a first, multi-window electronic aperture open;

FIG. 6B is an illustration showing an electronically masked imagingsystem with a second, multi-window electronic aperture open;

FIG. 7A is an illustration showing an addressable template patternsuitable for projection onto a surface of an object of interest;

FIG. 7B is an illustration showing an acquired image taken of a targetobject using an addressable template;

FIG. 7C is an illustration showing an acquired image and partial grid;

FIG. 7D is an illustration showing a reconstructed illustration of thecenter sample of FIG. 7C;

FIG. 8A is an illustration showing a non-laser pattern projector andimaging system;

FIG. 8B is an illustration showing a two prism off-set and two-sensorsystem;

FIG. 8C is an illustration showing a one silvered offset prism andtwo-sensor system;

FIG. 8D is an illustration showing a three CCD-sensor assembly system;

FIG. 8E is an illustration showing a narrow-band mirror sensor assemblysystem;

FIG. 9 is an illustration showing a laser pattern projector and imagingsystem;

FIG. 10 is a flow chart showing the acts of acquiring and processingimages in order to develop a representation of the surface of an object;and

FIG. 11 is a flow chart showing the incorporation of anaddressable-pattern to an imaging system in order to aid in imagereconstruction.

DETAILED DESCRIPTION

The present invention satisfies the long felt need for an inexpensiveand precise way of three-dimensional imaging (e.g., mapping). Aspects ofthe invention are applicable to surface and volume inspection ofmanufactured parts, comparing actual products versus the originaldesign, scanning of 3-D objects, evaluation of body parts (hernias,arteries, pre- and post-plastic surgery, etc.), surface roughnessevaluation, and real-time feedback of surface deformation. In thefollowing detailed description, numerous specific details are set forthin order to provide a more thorough understanding of the presentinvention. However, it will be apparent to one skilled in the art thatthe present invention may be practiced without necessarily being limitedto these specific details. In other instances, well-known structures anddevices are shown in block diagram form, rather than in detail, in orderto avoid obscuring the present invention.

The reader's attention is directed to all papers and documents which arefiled concurrently with this specification and which are open to publicinspection with this specification and, the contents of all such papersand documents are incorporated herein by reference. All of the featuresdisclosed in this specification, (including any accompanying claims,abstract, and drawings) may be replaced by alternative features servingthe same, equivalent or similar purpose, unless expressly statedotherwise. Thus, unless expressly stated otherwise, each featuredisclosed represents a non-limiting example of a generic series ofequivalent or similar features.

Furthermore, any element in a claim that does not explicitly state“means for” performing a specified function, or “step for” performing aspecific function, is not to be interpreted as a “means” or “step”clause as specified in 35 U.S.C. Section 112, Paragraph 6. Inparticular, the use of “step of” or “act of” in the claims herein is notintended to invoke the provisions of 35 U.S.C. 112, Paragraph 6.

First, an introduction to the present invention is provided to give anunderstanding of the general aspects. Next, defocusing methods based onlight properties and mask shape are discussed with respect to featuremapping. Then, aspects of single aperture systems are discussed withrespect to feature mapping. Subsequently, examples of pattern matchingare provided. Next, imaging methods according to the present inventionare provided. Next, a discussion of image matching is provided.

(1.0) Introduction

Blur from defocus can be used to measure the distance between a pointand the focal plane of a lens. The present invention proposes addeddimensions in terms of optical and illumination techniques to thesingle-lens multiple-aperture arrangement that overcome the shortcomingsof the original defocusing concept. The following aspects allow forrobust measurement of an object surface with a single-lens,single-sensor, and multiple-aperture device.

Optical modifications to the multiple-aperture arrangement physicallymask and convey filtered information to the sensor in such a way thateach aperture produces a separable image for reconstructing an objectsurface. In order to produce a separable image, the aperture mask may bemodified by altering the shape of the aperture, by coding thetransmittance of the aperture, or by providing a single-slit mask whosehole moves about the aperture plane during or between exposures. Each ofthe aperture masks provides additional information which aids inrepresenting the desired features of an object.

A single-lens, single-sensor, multiple aperture device may be furtheraugmented to obtain additional information from the object by usingregistered information. Registered information may be obtained from thedistinguishing characteristics of the object, from information projectedonto the surface of the object, or from information or markers placeddirectly onto the object.

For large objects which cannot be captured with a single exposure, theaforementioned aspects may provide information which may be used to fitmultiple exposures together in order to recreate surface features of adesired object. Alternatively, multiple images can be scanned in forboth large and small objects in order to produce a high resolutionrepresentation of the object or object feature. The matching concept isequally applicable to stereo vision systems.

Aspects of the present invention will now be described more fullyhereinafter with reference to the accompanying drawings, in whichpreferred embodiments of the invention are shown. This invention may beembodied in many different forms and should not be construed as limitedto the embodiments set forth herein. Further, the dimensions of layersand other elements shown in the accompanying drawings may be exaggeratedto more clearly show the details. The present invention should not beconstrued as being limited to the dimensional relations shown in thedrawings, nor should the individual elements shown in the drawings beconstrued to be limited to the dimensions shown.

(2.0) Light Property and Shape-Based Systems

A masked aperture generates a distinguishable image as light or otherelectromagnetic radiation from an illuminated object is passed through alens, through a masked aperture, and onto a sensor suitable forreceiving the information from the masked aperture. The masked aperturepasses coded and defocused information of the object onto a suitablesensor. The defocused information provides a measurement of a point onan object relative to the focal plane. The coded information from themasked aperture provides the information required in order to separateoverlapping images and match corresponding points detected by thesensor. Please note that although the term “light” may be used whendescribing various embodiments of the present invention, the presentinvention is suitable for use over any portion of the electromagneticspectrum, including but not limited to microwaves, infrared radiation,ultraviolet radiation, and X-rays. The use of the term “light” is forexemplary purposes and is not intended to limit the scope of the presentinvention to the visible portion of the electromagnetic spectrum.

When two or more masked apertures are used, each mask is ideallydifferent from the other such that the intensity versus wavelengthproperties and/or morphology of detected shapes from the maskedaperture(s) are easily distinguishable on the sensor. A variety offiltering apertures may be used in order to selectively filter lightaccording to its properties onto a light sensor such that the imagesfrom each aperture are distinguishable. Further, when the shapes of twoor more apertures are distinguishable, each aperture image detected bythe sensor is also distinguishable. Therefore, non-limiting examples ofsuitable aperture masks and filters include wavelength band-passfilters, light polarization filters, and differentially-shaped masks.

(2.1) Color Coded Filters

Referring to FIG. 1A, a band-pass filter system 100 is shown. Theband-pass filter system 100 includes a lens 102, a mask 104 having a redaperture 106 and a green aperture 108, and a sensor 110. Although shownas a red and a green aperture 106 and 108, respectively, any number andcombination of color filtered aperture may be used in combination withan appropriate sensor 110. Thus, while the apertures are referred tospecifically as the red and green apertures 106 and 108, respectively,the apertures are not intended to be limited to these colors and could,alternatively, be referred to as a first aperture, a second aperture,and so forth.

The band-pass filter system 100 produces a representation of anilluminated object 112 when the object 112 is placed in front of a focalplane 114. Scattered light 116 is reflected from the surface of theilluminated object 112 and through the lens 102. Once through the lens102, the scattered light 116 selectively passes through either the redaperture 106 or the green aperture 108, or is reflected off of orabsorbed by the mask 104. Transmitted red light 118 from the redaperture 106 and transmitted green light 120 from the green aperture 108are then recorded on the sensor 110 positioned in front of a focal imagepoint 122. As can be appreciated by one skilled in the art, the color oflight used to illuminate the object can also be selected such that itonly passes through a desired aperture or set of apertures. Use ofnarrow-band light projectors can be useful in situations where one setof apertures is used to capture defocusing information in one color,while another aperture is used to project a realistic visual image ofthe object in another color, so that the two are readilydistinguishable.

Referring to FIG. 1B, a defocused, multiple color-coded imageacquisition 124 of real points is shown as received by the sensor 110 ofFIG. 1A. Each color-coded acquisition 124 corresponds with amulti-wavelength addressable-pattern created by the respective aperture106 and 108. As shown in FIG. 1B, each real point on the object isrepresented with multi-wavelength-addressable-pattern red dots 126 andgreen dots 128. As can be appreciated by one skilled in the art, the redand green dots 126 and 128 are a result of the red and green apertures,respectively; however, the invention is not limited thereto as the colorof the dots would vary according to the color of the apertures.Corresponding red dots 126 and green dots 128 are shown linked togetherwith a correspondence line 130. The correspondence lines 130 are notvisible; however, they are useful tools for highlighting the difficultyof matching points in color-coded image acquisitions 124. Only the dotsconnected by correspondence lines 130 actually correspond together.Without the mask 104, there would not be enough information to linkcorresponding points.

Referring to FIG. 1C, an enhanced view of the framed area 132 of FIG. 1Billustrates the procedure for matching a corresponding red dot 126 andgreen dot 128. When the multiple color-coded image acquisition 124 hasbeen developed, a processor then begins a search for all of the colorcoded dots within the image. Alternatively, the search may be conductedfrom raw data (i.e., an actual image 124 need not be produced). Instead,the sensor 110 is coupled with a processor which receives the sensorinformation directly. In either case, once all dots have been detected,the matching process begins with an assumption of the relative positionof the illuminated point 112 with respect to the focal plane 114. Therelative position of the illuminated point 112 with respect to the focalplane 114 may be known a priori, entered by a user, determined bysoftware, or determined by sensors. For illustrative purposes, here itis postulated that the illuminated point 112 of the object is in frontof the focal plane 114. Therefore, the matching begins with theinstruction command, for example: “Any green dot 128, 136, 138, and 140to the right of a red dot 126, 142, 144, and 146 on a line correspondingto a line connecting the two apertures (within a tolerance) is a match.”The first red dot 126 is detected, and then matched to the first greendot 128 within tolerance 134 of the red dot 126 according to theinstruction command. The tolerance 134 in this case is denoted as adistance from the red dot 126 in the form of a radius. However, thetolerance 134 may take the form of any desired shape or distance.Supplemental searches conducted for green dots 136, 138, and 140 withinthe tolerance 134 of the red dot 126 produces a total of three “ghost”matches (green dots 136, 138, and 140, respectively).

Referring to FIG. 1D, the relationship of focal length (L) to Z-distanceof matches and “ghost” particles with respect to FIG. 1C is shown. Thematching of the red dot 126 to all of the green dots 128, 142, 144, and146 results in one match 148 and three ghosts 150, 152, and 154. Thematch between the red dot 126 and the green dot 128 is used to calculatethe Z-to-L relationship of the first matched point 148. The mismatchbetween the red dot 126 and the green dots 136, 138, and 140 providesthe first three ghosts 150, 152, and 154, respectively.

With respect to the second red dot 142, one match 156 and two ghosts 158and 160 are produced. The match between the second red dot 142 and thecorresponding green dot 136 is used to calculate the Z-to-L relationshipof the second matched point 156. The mismatch between the red dot 142and green dots 138 and 140 is represented by the two ghosts 158 and 160respectively.

With respect to the third red dot 144, one match 162 and two ghosts 158and 160 are produced. The ghosts 158 and 160 are dots that are notassignable to a corresponding dot from the other aperture. The matchbetween the third red dot 144 and the corresponding green dot 138 isused to calculate the Z-to-L relationship of the third matched point162. The single mismatch between the red dot 144 and green dot 140 isrepresented by the ghost 164.

Finally, with respect to the fourth red dot 146, one match 162 but noghosts are generated. The match between the fourth red dot 146 and thecorresponding green dot 140 is used to calculate the Z-to-L relationshipof the fourth and final matched point 166. Since there are no othergreen dots to the right of the red dot 146 other than the matching greendot 140, no additional mismatches exist for the framed area 132 of FIG.1C.

Determining the Z-to-L relationship between matches and “ghost”particles is greatly enhanced by differentially-coded points, such asthose shown 126 and 128 in FIG. 1B. In a non-separable case, one inwhich there is no color information provided by an aperture mask 104,there are many more ghosts because, without having a differentiator likecolor, each “red dot” of FIG. 1A can be matched with any other “red dot”producing many more ghosts. Further, no assumptions can be made that anygiven dot by itself is not, in fact, two dots on top of the other,adding even more ghosts at the focal plane.

(2.2) Polarized Filters

Please note that although the term “light” may be used when describingvarious embodiments of the present invention, the present invention issuitable for use over any portion of the electromagnetic spectrum,including but not limited to microwaves, infrared radiation, ultravioletradiation, and X-rays. The use of the term “light” is for exemplarypurposes and is not intended to limit the scope of the present inventionto the visible portion of the electromagnetic spectrum.

Coded information may be provided to a sensor in any number of ways. Asa non-limiting example, FIG. 2A illustrates a polarized filter imagingsystem 200. The polarized filter imaging system 200 includes a lens 202,a mask 204 having a horizontal polarizing aperture 206 and a verticalpolarizing aperture 208, and a sensor 210 capable of distinguishingbetween polarizations. Although shown as a combination of horizontallyand vertically polarized apertures 206 and 208 respectively, any numberand combination of at least nearly orthogonal pairs of orientations maybe used.

The polarized filter imaging system 200 produces a representation of theilluminated object 212 when placed in front of the focal plane 214.Scattered light 216 is reflected from the surface of the illuminatedobject 212 and through the lens 202. Once through the lens 202, thescattered light 216 selectively passes through either the horizontalpolarizing aperture 206 or the vertical polarizing aperture 208, or isreflected off of the mask 204. The transmitted horizontally polarizedlight 218 from the horizontal polarizing aperture 206 and thetransmitted vertically polarized light 220 from the vertical polarizingaperture 208 is then recorded on the sensor 210 positioned in front ofthe focal image point 222.

By differentially coding the horizontal polarizing aperture 206 and avertical polarizing aperture 208, distinguishable dots, similar to thoseshown in FIG. 1B, are obtained. However, the coded information obtainedfrom the present polarized aspect provides polarization markers insteadof color-coded dots.

A similar result can be obtained by using at least onepolarization-coded aperture as shown in FIG. 2B, where if the at leastone aperture is rotated from a first aperture position 224 to a secondaperture position 226 with an exposure taken at each position, thepolarization of the aperture will change between exposures, resulting inmutually distinct sets of polarized images 228 and 230 from the firstexposure 228 and the second exposure 230 respectively, whereby the depthinformation can be determined by measuring the distance between images228, 230 from the same marker 232 on different exposures.

Selectively transmitting light (as is the case with a band-pass filtersystem 100) or exploiting properties of light (as is the case with apolarized filter imaging system 200) are effective means of codinginformation received by a sensor. Ultimately, the coded informationdetected by the sensor eases the matching task described with respect toFIG. 1C and FIG. 1D.

(2.3) Spatially-Biased Apertures

Referring to FIG. 3A, a differentially-shaped aperture system 300 forimaging points small enough to be considered nearly point sources, isshown. The differentially-shaped aperture system 300 includes a lens302, a mask 304 having a circular-shaped aperture 306 and asquare-shaped aperture 308, and a sensor 310. Although shown as acircular-shaped aperture 306 and a square-shaped aperture 308, anynumber and combination of different shape-filtered apertures may beused. Non-limiting examples of suitable shapes include convexpolyhedrons, concave polyhedrons, circular shapes, polyforms, andcombinations thereof.

The differentially-shaped aperture system 300 produces tworepresentations 314 and 316 of the illuminated object 312 per exposure.Each shape 314 and 316 detected by the sensor 310 corresponds to theshape of the respective aperture 306 and 308, respectively. As scatteredlight 320 is reflected off the surface of the illuminated object 312 andthrough the lens 302, it will either pass through the circular-shapedaperture 306, the square-shaped aperture 308, or be reflected by themask 304 and beyond the sensor focal plane 318. The transmitted light322 which passes through the circular-shaped aperture 306 produces acircular pattern 314 on the sensor 310. Similarly, the transmitted light324 which passes through the square-shaped aperture 308 produces asquare pattern 316 on the sensor 310. After multiple acquisitions, thenumerous circular patterns 314 and square patterns 316 are detected andthen matched by a processor 326 based upon a matching rule. Both thematches and ghosts may then be plotted on a Z-to-L plot, such as the onedepicted in FIG. 1D. Alternatively, a plot demonstrating the matcheswithout ghost images may also be generated.

In addition to apertures of different shape, spatially-biased aperturescan also comprise similarly shaped apertures 326 and 328 located atdifferent radial positions from the center of the mask 329, as shown inFIG. 3B. When this arrangement of apertures is rotated from a firstposition 326 and 328 to a second position 330 and 332 and an exposure istaken at each position (sequential time-delayed imaging), the distanceof the aperture from the center of the mask 329 will determine the ratewith which images 336 and 338 produced by an object change theirposition on the imager 334, where the rate of change physicallymanifests as the distance the image moves between exposures.

Another embodiment of spatially-biased apertures suitable for use withthe present invention are apertures of similar shape but different size,for example, two circular apertures, where one is larger than the other.Using apertures of different size effectively performs the same functionas using apertures of different shape, as described above and shown inFIG. 3A.

(3.0) Single Aperture System

Referring to FIG. 4A and FIG. 4B, a synced single-aperture system 400including a lens 402, a single-hole mask 404, a moving aperture 406, asensor 408, and a processor 410 in communication with the sensor 408, isshown. Additionally, the single-hole mask 404 is shown in a firstposition A and a second position B, respectively. An illuminated object412 may be reconstructed by selectively allowing reflected rays 414 topass through the lens 402 and the aperture 406 of the single-hole mask404. The position of the single-hole mask 404, whose moving aperture 406moves about the aperture plane between exposures, is recorded by theprocessor 410. As shown in FIG. 4A, the moving aperture 406 transmitslight 416 and produces a first point 414 detected by the sensor 408. Thefirst position information of the moving aperture 406 during the firstexposure is recorded by the processor 410 as shown in FIG. 4A. For thesecond exposure, the moving aperture 406 is moved to the second positionB (shown in FIG. 4B). As shown in FIG. 4B, the moving aperture 406transmits light 418 and produces a second point 420 detected by thesensor 408. The second position information of the moving aperture 406during the second exposure is recorded by the processor 410. The firstpoint 414 and first position information and second point 420 and secondposition information are then used to match the first point 414 from thefirst exposure with those of the second point 420. Alternatively, thecolor of the reflected rays 414 may be altered between the firstexposure and second exposure in order to provide additional informationwhich may be used to aid in the matching process.

Similarly the problem of mismatching can be alleviated by rotating theaperture 422 as shown in FIG. 4C. When a plurality of image acquisitionshave been taken relative to the oblong aperture by rotating the apertureover time, overlapping images can be distinguished, thereby alleviatingthe ambiguity generated by image overlap. The figure shows a comparisonof the images formed by two objects 424 positioned in a horizontal planewith the aperture at a first aperture position 422. In the firstaperture position 432 the objects' images 426 overlap, causing apotential mismatch. When the aperture is rotated to a second apertureposition 428, however, the images formed are distinguishable 430.

(3.1) Single Slit-Aperture System

Referring to FIG. 5A, a single-aperture system 500 having multiplef-stops is illustrated. The differentially-shaped aperture system 500includes a lens 502, a mask 504 having a substantially oblong aperture506, and a sensor 508, and a processor 510 in communication with thesensor 508. Although shown as a roughly oblong-shaped aperture 506, ingeneral, any aperture which is significantly longer in length than inwidth may be used regardless of shape.

An illuminated object 512 may be reconstructed by selectively allowingreflected rays 514 to pass through the lens and the substantially oblongaperture 506 of the mask 504. Notably, the single-aperture system 500uses a long, narrow, slit-aperture 506, instead of a standard circularaperture. Effectively the slit aperture 506 has a different f-number intwo directions. The long length of the slit aperture 506 produces a lowf-number which generates a large variance disc 516 on the sensor 508.Conversely, the narrow width of the slit aperture 502 produces a highf-number, generating a minimum variance, such that the image of a pointsource is represented by lines 518 rather than discs 516. The intensitycan now be thought of as varying inversely with length rather than area,so the dynamic range required on the sensor is much decreased relativeto a pure-blur system. Further, the size of the produced images 516 and518 only increase in one direction, minimizing the chance for overlap.

Also, the slit aperture could be made to be asymmetric in shape 542 asshown in FIG. 5F. The purpose of the asymmetry is to allow the sensor544 to determine whether an object is located in front of 546 or in backof 548 the focal plane 550. An object located in back of 548 the focalplane 550 will produce an inverted image 552 on the sensor 544, while anobject located in front of 546 the focal plane 550 will produce a normalimage 554. However, if the aperture is symmetrically shaped 506 as inFIG. 5A, the image produced 516 by an object in back of 548 the focalplane 550 will be indistinguishable from one located at thecorresponding location in front of 546 the focal plane 550. By using anasymmetrical aperture 542, these objects in front of 546 and in back of548 the focal plane 550 can be distinguished. The asymmetric aperture542 shown in FIG. 5F has a circular hole at one end of the overalloblong shape, but any asymmetrically shaped aperture will produce thesame effect.

Referring to FIG. 5B, an image 520 acquired from the sensor 508 of asingle-aperture multiple f-stop system 500 is shown. Within the frame522 of the image 520, multiple plots 524, 526, 528, and 530 withdifferent Z-coordinates are shown. Although shown as an image 520, theinformation depicted may also be conditioned and sent via a signal to aprocessor 510 for processing.

Referring to FIG. 5C, the framed area 522 of the acquired image of FIG.5B is processed in order to find the multiple f-stop streaks 524, 526,528, and 530 corresponding with the aperture movement. Once all of themultiple f-stop streaks 524, 526, 528, and 530 have been found, a ruleis applied to determine the Z-to-L relationship. Notably, no matching isrequired.

By assuming all of the points were in front of the focal plane “L,” themultiple f-stop streaks 524, 526, 528, and 530 are used to calculate theZ-to-L relationship. An example of matched points 532, 534, 536, and 538determined by a processor 510 are shown in FIG. 5D. In general, theprocessor 510 connected with the sensor 508 may be used to collect theraw data obtained from the sensor. The processor 510 then may use theZ-to-L relationships in order to calculate the depth information of eachdetected f-stop streaks 524, 526, 528, and 530. The processor 510 maythen be used to generate a representation of the object from the depthinformation of each illuminated point 512. In another aspect, theprocessor 510 may also include memory. The memory may be used to storecalibration information of previously sampled points at known distances.The calibration information may be stored as a look-up table in theimage-acquisition system for fast in-line processing. Alternatively, thecalibration information may be stored remotely and accessed by theprocessor.

The results depicted in FIG. 5B, FIG. 5C, and FIG. 5D may also beobtained by using a vibrating, single-aperture system 540 such as theone illustrated in FIG. 5E. The vibrating, single-aperture system 540includes a lens 502, a mask 504 having a single moving aperture 506, anda sensor 508.

An illuminated object 512 may be reconstructed by selectively allowingreflected rays 514 to pass through the lens and the substantially oblongaperture 506 of the mask 504. Notably, the single-aperture system 500uses a moving aperture 506, effectively simulating the effect of havinga different f-number in two directions. As the moving aperture 506controllably oscillates right to left in the direction of A and B (or inany other suitable direction), the net displacement of the movingaperture 506 from A to B produces a low f-number. The low f-number ofthis lateral movement from A to B generates a large variance disc 516 onthe sensor 508. Further, as the moving aperture 506 moves from A to B,there is no net change to the vertical diameter of the moving aperture506. The constant height of the moving aperture 506 therefore produces ahigh f-number, generating a minimum variance, such that the image of apoint source is represented by lines 518 rather than discs 516. Theintensity is dependent upon the amount of time the aperture 506 spendsat a particular axial position, thus image generated by this techniquelook more like bright ends connected by dimmer straight lines. Further,the size of the produced images 516 and 518 only increase in onedirection, minimizing the chance for overlap.

In one aspect, the invention can be thought of as a two-aperture systemwith the ambiguity of matching removed by simply connecting the twoaperture images physically on the imager. When imaging large objectsthrough the aperture (not point sources), three images are visible. Thecenter image is the image of the object, and the outer two images areformed as a result of diffraction and lens effects. As the scale of theobject decreases, it approaches a point source, and at the limitingcase, the image of the point source object has the same shape as theaperture.

(3.2) Large Central Aperture with Off-Axis Defocusing Apertures

Please note that although the term “light” may be used when describingvarious embodiments of the present invention, the present invention issuitable for use over any portion of the electromagnetic spectrum,including but not limited to microwaves, infrared radiation, ultravioletradiation, and X-rays. The use of the term “light” is for exemplarypurposes and is not intended to limit the scope of the present inventionto the visible portion of the electromagnetic spectrum.

The problem of mismatching can also be alleviated by using a largecentral aperture 556 in conjunction with at least one off-axisdefocusing aperture 558 as shown in FIG. 5G. The central aperture 556can be a central variable aperture as in a typical camera. Effectively,the central aperture 556 has a different f-number than the defocusingapertures 558. This means that at all times a substantially full image560 of the object 561 is present on the sensor 562 via the centralaperture 556 with superimposed defocused dots 564 from the defocusingapertures 558.

Having the object image available at all times serves three purposes.First, it allows an operator to see where the device is pointed. Second,the object image provided by the central aperture can be matched withthe physical x-y-z locations of points imaged by the defocusingapertures to produce a map of the object surface (see section 4.0“Pattern Matching” below). Finally, it allows an accurate estimate of“POISE” (position and orientation) from two-dimensional (2-D) imagesproduced by the device with respect to the object. Various methods fordetermining “POISE” are well known in the art. Existing “POISE” methodscan use features of the background image or pre-positioned marked pointsto make their estimations. When using features of the background image,camera position can be obtained by proper rotation, de-warping, andscaling of the actual image obtained by the central aperture fromdifferent camera positions. In the case where a light projection systemis used, the points of light projected onto the object and imagedthrough the central aperture can be used to obtain camera position.Examples of suitable “POISE” methods for use with the present inventioncan be found in U.S. Application Publication No. 2007/0103460A1 to Zhanget al., titled “Determining Cameral Motion;” U.S. Patent ApplicationPublication No. 2007/0008312A1 to Zhou et al., titled “Method forDetermining Camera Position from Two-Dimensional Images that form aPanorama;” International Application No. PCT/US2006/060724 to 3MInnovative Properties Company, titled “Determining Camera Motion;” andLowe, David G, “Three-Dimensional Object Recognition from SingleTwo-Dimensional Images,” Artificial Intelligence, 31, 3 (March 1987),pp. 355-395.

While the central aperture provides valuable information for positioningpurposes (“POISE”), it may cause overcrowding. In one embodiment of thepresent invention, and as shown in FIG. 5H, the central aperture 556with off-axis defocusing apertures 558 arrangement is used inconjunction with a light projector 564 for projecting a predeterminedpattern of light 566 onto the surface of an object 561. Thepredetermined pattern 566 is defocused through the defocusing apertures558, and the amount of defocus in the pattern image 568 is used todetermine depth information about the object 561. A potential drawbackof this configuration is that the central aperture will also produceunshifted (non-defocused) images of all the points in the projectedpattern, which may interfere with the defocused points produced by thedefocusing apertures. By using a dot projection system with anarrow-band light source in conjunction with an optical filter on thecentral aperture 556 (represented as horizontal lines) for selectivelyfiltering out the wavelength of projected light, the unshifted imagesproduced by the central aperture can be removed. In addition, aconjugate filter on the defocusing apertures 558 (represented byvertical lines that allows only the wavelength of projected light topass can be used to stop the object's image from forming through theoff-axis defocusing apertures, thus keeping the object image 560 fromgetting blurred.

Further, when using a plurality of defocusing apertures, if thedefocusing apertures are placed asymmetrically with respect to thecentral aperture, then the images of points will also contain thisdistinction, and the orientation of the image indicates whether theforming point was ahead of or behind the focal plane of the lens. Thistechnique performs the same function as using a single asymmetricalaperture as previously described.

Finally, while the addition of a central aperture can provide helpfulreference information in a system with two defocusing aperture system,there is no limit to its application to systems with three or moredefocusing apertures.

(3.3) Electronic Masked Aperture

Referring to FIG. 6A and FIG. 6B, an electronically masked imagingsystem 600 is shown, including a lens 602, an aperture plate 604, amulti-window electronic aperture 606, a sensor 608, and a processor 610in communication with the sensor 608 and aperture plate 604.Non-limiting examples of suitable aperture plates 604 include a liquidcrystal display (LCD) which may be fully synchronized with the sensor608. In one aspect, the sensitivity may be controlled by varying the“off-axisness” of the apertures. An illuminated object 614 may bereconstructed by selectively allowing reflected rays 616 to pass throughthe lens 602 and one of the many windows of the multi-window electronicaperture 606.

As shown in FIG. 6A, a first window 612 of the multi-window electronicaperture 606 transmits light 618 and produces a first point 620 detectedby the sensor 608. During the first exposure, the first open window 612position information is recorded by the processor 610.

To obtain a second exposure, a second window of the multi-windowelectronic aperture 606 is opened. As shown in FIG. 6B, a second window622 of the multi-window electronic aperture 606 allows light 624 to passand produces a second point 626 detected by the sensor 608. During thesecond exposure, the second open window 622 position information isrecorded by the processor 610. The first point 620 and first open window612 position information and second point 626 and second position openwindow 622 position information are then used to match the first point620 from the first exposure with the information of the second point626.

(4.0) Pattern Matching by Pattern Projection

If the object of interest is a surface whose shape is to be matched, apredetermined pattern of markers may be projected on the surface and thepoints in the detected image may be sampled by measuring the relativeposition of the projected markers. The sensor's allowable upper densitylimit of imaged dots is the imaging system's limit. Once the points areidentified in each view, there is only the question of whether the samepoint exists in both views. In another aspect, if the volume to bemapped contains a cluster of asymmetrical cells in a volume, then theshape and orientation of the cells can be used as an additionalconstraint in the inter-view matching, thus reducing the chance that amismatch can occur. This aspect is referred to as “feature matching.”

Referring to FIG. 7A, an illustration of an addressable template pattern700 suitable for projection onto the surface of the object of interestis shown. The addressable template pattern 700 is projected orphysically placed on the target surface and then captured by an imagingsystem at varying distances (Z) from the object. In one aspect, theaddressable template pattern 700 is in the form of a grid pattern with adistinguishable center point 702.

Referring to FIG. 7B, an illustration of an acquired image 704 taken ofa target object using an addressable template is shown. As illustrated,some dots 706, 708, 710, and 712 are missing in the acquired image 704.

Referring to FIG. 7C, the acquired image 704 with a partial grid 714 isshown. If the addressable-pattern 702 is ordered, a grid pattern 714with a distinguishable origin can employ a method such as “structuredpattern matching” to reduce the number of required viewpoints, or imageacquisitions, to two. The addressing algorithm, such as one stored on acomputer-readable medium or executed by a processor, processes eachaperture's image to find the relative address of each dot in theaddressable-pattern 704 according to the template pattern. Anon-limiting example of a suitable addressable template pattern 700 isillustrated in FIG. 7A. The addressing algorithm has some tolerance toallow for deformation of the addressable-pattern 704 (See FIG. 7B andFIG. 7C). The deformation of the addressable-pattern 704 is noticeablewhen contrasted with the original addressable template pattern 700 (SeeFIG. 7A). Further, the addressing algorithm can also account for missingentities 706, 708, 710, and 712 in the acquired image 704. Missinginformation is considered missing when a point on the addressabletemplate pattern 700 does not appear in the addressable-pattern 704.

A reconstructed illustration of the center sample 716 of FIG. 7C isillustrated in FIG. 7D. The points are reconstructed by calculating theZ for each pair of dots with the same address. Any pair with a missingdot is not reconstructed.

(4.1) Pattern Projector (Non-Laser)

Please note that although the term “light” may be used when describingvarious embodiments of the present invention, the present invention issuitable for use over any portion of the electromagnetic spectrum,including but not limited to microwaves, infrared radiation, ultravioletradiation, and X-rays. The use of the term “light” is for exemplarypurposes and is not intended to limit the scope of the present inventionto the visible portion of the electromagnetic spectrum.

Referring to FIG. 8A, a non-laser pattern projector 800 and imagingsystem 802 are shown. The non-laser pattern projector 800 comprises alens 804 identical to the imaging lens 806 of the imaging system 802.The lens 804 of the non-laser pattern projector 800 is placed at anequivalent distance from the beamsplitter 808 as the lens 806 of theimaging system 802. This causes the principal rays 810 of the projectedpoints 812 to coincide with the principal rays 814 detected by thesensor 816 of the imaging system 802. Thus the projected pattern 818will look as though it does not move in the detected image, even whenthe distance between the projected point 812 and the focal plane 820 ofthe imaging lens 806 changes. This makes identifying anaddressable-pattern 818 much easier, even if some points (e.g., dots)are missing.

The prerequisite is that the images from each viewpoint are physicallyseparate—this is naturally true in multiple-sensor systems such asphotogrammetry, but requires special care with systems like thedefocusing concept (multiple apertures on a single lens imaging onto asingle sensor).

The projected pattern 818 is produced by passing light 822 through apattern stencil 824 and projector lens system 826 with a lens 804substantially identical to the imaging lens 806.

For single-lens systems, the aperture images must be separate. This canbe accomplished with prisms (see FIGS. 8B and 8C) or fiberoptic bundlesso that each aperture projects onto a separate sensor, or with aphysically masked aperture (see FIGS. 8D and 8E) if the sensor is acolor sensor.

Referring to FIG. 8B, a two prism off-set and two-sensor system 828 isshown. The system 828 comprises a first prism 830, second prism 832, anda first sensor 834 and second sensor 836 behind a mask and two-slitaperture 838. The first prism 830 and second prism 832 offset theincoming light 840 and 842 from the two-slit aperture 838 such thatlight transmitted through the first prism 830 and second prism 832 maybe detected by separate sensors 834 and 836. Such a configuration may beused when the two-slit aperture 838 is used to code information based onthe inherent properties of light or the light must be separated as isthe case when addressable-pattern techniques are employed. Non-limitingexamples of suitable inherent properties include but are not limited tothe frequency, frequencies, or polarization of coded transmitted lightdetected images.

Referring to FIG. 8C, a one silvered offset prism and two-sensor system844 is shown. The system 844 comprises a silvered prism 846, a firstsensor 848 and second sensor 850 behind a mask and two-slit aperture852. The silvered prism 846 offsets the first bundle of incoming light854 from the two-slit aperture 852 such that light transmitted throughthe silvered prism 846 may be detected by the first sensor 848.Alternatively, light 856 which has passed through two-slit aperture 852may also be detected separately on the second sensor 850.

Referring to FIG. 8D a three CCD-sensor assembly system 858 is shown.The system 858 comprises a three CCD-sensor 860 behind a mask andtwo-slit aperture 862. The CCD-sensor 860 includes a blue sensor 862, agreen sensor 864, and a red sensor 866. The system of prisms 868 offsetsthe first bundle of incoming light 870 from the two-slit aperture 856such that light transmitted through the prism 868 may be detected by thered sensor 866. Alternatively, light 872 which has passed through thetwo-slit aperture 852 may also be detected separately on the greensensor 864.

FIG. 8E is a narrow-band mirror sensor assembly system 874 is shown. Thesystem 874 comprises a narrow-band mirror 876, located behind a mask andtwo-slit aperture 878, and a first sensor 880 and second sensor 882. Thesystem narrow-band mirror 876 offsets the first bundle of incoming light884 from the two-slit aperture 878 such that light transmitted throughthe narrow-band mirror 876 may be detected by the first sensor 880.Alternatively, light 886 which has passed through the two-slit aperture878 may be detected separately on the second sensor 882.

(4.2) Pattern Projector (Laser)

Any lens can be represented by two “principal planes.” The location ofthe planes is only a function of the lens, and all principal rays (whichdefine the image centerline for a point) behave as if they entered thefirst principal plane and exited the second principal plane at the axis.

By using measurements of the location of the front principal plane andthe field of view, a diffraction grating with the desired pattern can bemade and positioned such that the beams from the laser projectorcoincide with the principal rays of the imaged dots. Thus, the projectedpattern will look as though it does not move in the image even when thedistance between the projected dot and the focal plane of the imaginglens changes. This makes searching for the addressable-pattern mucheasier even if some dots are not imaged.

A complex ray trace through a compound lens (where the ray kinks atevery air/glass interface) can be mathematically represented as twoplanes at which the rays kink. Thus, the left image shows the “real” raytrace, and the right image shows the mathematical representation of suchlens. The planes are found by taking any chief (also called principal)ray coming into the first glass interface and leaving the last glassinterface and extending them to intersect the axis of the lens. Theintersection marks the location of the planes.

Thus, one would first do a calibration (by imaging a grid at severalZ-distances) and then do a least-squares type fit to find out wherethose two planes are, and what the field of view angle is. Then, thediffraction grating can be customized to match the field of view angle,and put at the same distance from the beam-splitter as the firstprincipal plane. Therefore, the laser beams will follow the exact pathof the principal rays.

In operation, as an object gets closer to a lens, it appears larger inthe image. This means that the edges of the object move laterally on theimage. The same would be true of any pattern projected in front of thecamera onto a surface. By making the rays match exactly as in the aspectdepicted in FIG. 9, none of the points ever move laterally, regardlessof their Z-position.

Now, if a two-hole aperture mask is added, the corresponding dots stillmove apart from each other (the dots (images) are formed by the marginal(outer rays). However, since the principal ray is not moving laterally,the centroid of the corresponding “match shape” will not move laterally.Conceivably, once the distinguishable dot of the addressable-pattern islocated, the centroid of that match can be found. Knowing that thepattern is never expanding laterally, it is known where the centroid ofevery other point on the pattern should be, which should aid in“addressing” the points.

This is different than the traditional addressable-pattern search, wherethe points are all moving relative to each other, so that if there's toomuch of a surface Z-change, the pattern may not be reconstructible.

Referring to FIG. 9, a laser pattern projector system 900 and imagingsystem 902 are shown. The laser pattern projector system 900 comprises alaser projector 904 and a filtering mask 906. The filtering mask 906selectively passes light from the projector 904 onto the fifty percentbeam splitter 908. The laser projector 904 and a filtering mask 906 arein-line with the beamsplitter 908 which causes the principal rays 910 ofthe projected points 912 to coincide with the principal rays 914detected by the sensor 916 of the imaging system 902. Thus the projectedpattern 918 will look as though it does not move in the detected image,even when the distance between the projected point 912 and the focalplane 920 of the imaging lens 906 changes. This makes identifying anaddressable-pattern 918 much easier, even if some points (e.g., dots)are missing.

(5.0) Imaging Methods

Referring to FIG. 10, a flow chart depicting the steps of acquiring andprocessing images to develop a two dimensional or three dimensionalrepresentation of the surface of an object is shown. Any single-lensdevice may be built or modified to include an imaging lens, an apertureconfigured to generate distinguishable images, a sensor, and aprocessor.

The imaging process begins by illuminating the surface of the object1000. The surface may illuminated by the imaging system or a suitableexternal lighting source. Light is reflected off of the surface of theobject and transmitted through the aperture 1010. The aperture may beplaced in the plane of the imaging lens, in front of the imaging lens,behind the imaging lens, may be applied at an aperture plane of theimaging lens when accessible, or made accessible via a relay lenssystem.

As the light travels past the aperture, the aperture may be used in anumber of ways to code information received by the sensor. Non-limitingexamples of suitable methods by which light may be coded in order toproduce distinguishable images 1020 on the sensor include but are notlimited to: filtering transmitted light according to lightcharacteristics (such as filtering by wavelength or polarization),transmitting light as a function of time such that the distinguishableimages are allowed to pass through the aperture as a function of time;or physically altering the shape of the aperture to comprise a series ofdifferent shapes from which transmitted light through the apertureproduces distinguishable shape-based images.

An act which aids the system in determining whether or not to acquireadditional images 1030 can also be implemented. The act may further beaugmented to weigh the suitability of an acquired image. For example, animage detected by a sensor which suffered from excess movement duringthe exposure may be discarded by the algorithm. In this case, the lastacquired image is discarded and the process is re-acquired with theillumination of the object 1000. In another aspect, the received imagemay be suitable for processing; however, more image acquisition imagesare needed 1030. In this case, a further decision can be added tofurther augment the algorithm, an example of which would be to add adecision to determine whether or not the viewpoint of the imaging systemshould be adjusted 1040. If the position of the imaging device or thedesired area of the object needs to be shifted, either the imagingsystem or the target object may be altered to adjust the viewpoint 1050.

Once all or at least some of the images have been acquired, therelationship amongst points, or point information, within each image isused to calculate or determine the relative or absolute distanceinformation for each point 1060. Once the distance information is known,the information may be fed to an algorithm which uses the distanceinformation to generate a representation (e.g., 3-D mapping) of theobject 1070.

(6.0) Image Matching

For large objects or applications which require multiple exposureacquisitions, image matching provides a method by which related imageacquisitions may be tied together to recreate an object surface.Although not required to recreate the target object, when the positionof the imaging system is known relative to the target object, imagematching offers the ability to recreate a target object with exactmeasurements. In general, image matching, also referred to as digitalquilting, is greatly aided by the use of an addressable-pattern templateimage. In one aspect, the addressable-pattern projector may bephysically tied to the acquisition device. In another aspect, theaddressable-pattern projector may move independently of the device, butin such a way that the pattern visible by the device is stilladdressable.

An imaging device acquires an addressable-pattern template image at aninitial position. The addressable-pattern template image typically has afixed number of points in the X, Y, and Z-planes. The position of theimaging device is then adjusted and a second addressable-patterntemplate image is acquired at second position. Precautions may be takensuch that adjusted positions determined to exceed motion constraints areignored. The second position, or adjusted position, is related to theinitial imaging device position by a six-variable solid translation androtation. Typically, the adjusted position is related to the initialposition by the fact that the image captured at the new positionoverlaps in part with the first template image and has a substantiallysimilar number of points.

In operation, at least one outer hull is generated by a processor or ismanually highlighted by the user. The outer hull encompasses all thepoints within the addressable-pattern template image andaddressable-pattern surface image. Although not always the case, thepoints outside the addressable-pattern template image outer hull may bedisregarded. A plurality of inner hulls of the points in theaddressable-pattern surface image is also generated. The inner hull is afunction of a maximum acceptable displacement between acquisitionswithin the intersection of the plurality of hulls, according to thesix-variable solid-body translation and rotation. The error may becalculated from the difference between a point on theaddressable-pattern surface image and the addressable-pattern templateimage.

When the hulls have been generated, the addressable-pattern informationis processed using a matching algorithm. The matching algorithm isconfigured to determine the distance between each point on theaddressable-pattern surface image and its corresponding point on theaddressable-pattern template image. Each of the matched points is thenformed from the plurality of inner hulls according to their solid-bodytranslations and merged with rotations to form a high-resolution dataset.

When hundreds or possibly thousands of acquisitions have been matched,the well-defined point clouds are merged according to their solid-bodytranslations and rotations. An algorithm that uses theaddressable-pattern information may also be adapted to determine whetheror not enough matching points exist to recover the features of thetarget object. When a well-defined point cloud has been developed, thehigh-resolution point cloud can be used to generate or output ahigh-resolution surface (nurbs, meshes, etc.) with or withoutinterpolation via standard algorithms or commercial packages, such asGeomagic Studio. Geomagic Studio is produced by Geomagic, located at3200 East Hwy 54, Cape Fear Building, Suite 300, Research Triangle Park,N.C., 27709 U.S.A.

The fit is considered satisfactory if the total error is below somethreshold which is a function of the precision of the device. Once thisis done, a second acquisition at the adjusted position becomes thetemplate and the next acquisition becomes the surface matched to it. Therobustness of addressable-pattern information in the matching algorithmallows for the matching of small set to small set, without interpolatingthe surface shape until enough acquisitions are available.

FIG. 11 is a flow chart depicting the use of an addressable-pattern toaide in image reconstruction. The use of an addressable-pattern duringsurface feature acquisition is one way of bypassing the correspondencesearch employed in a separable-viewpoint three-dimensional imagingsystem.

After the starting process 1100 begins with the acquisition of a numberof images, each of the images containing (being illuminated with) anaddressable-pattern 1102. Each image is typically taken from a differentviewpoint, although the addressable-pattern is static with respect tothe contours of the object's surface. Each of the plurality of imagescomprises at least a portion of the addressable-pattern information andat least one point represents at least one aspect of the target object.It will be appreciated by one of skill in the art that an object mayinclude a variety of points on the object. Each point may provideimportant information with respect to the eventual reconstruction of theobject.

An address is assigned to each point in the image in an addressing act1110. In general, the addressable-pattern provides a sequence or seriesof plots on the object which may be referenced to assist in theaddressing act 1110. Importantly, the addressable-pattern need not besymmetrical or contain a regular sequence of markers or images.Non-limiting examples of suitable addressable-pattern information mayinclude a color sequence pattern, a pattern comprising differentlyshaped object, a position sequence pattern, distinguishable objectfeatures or object landmarks, or any combination thereof. Theaddressable-pattern image may be placed on the surface of the object ina variety of ways. Non-limiting examples of suitable methods include:projecting the addressable-pattern image onto the surface of the object;physically placing an addressable-pattern image onto the surface of theobject; and using the features inherent to the object being imaged as asource.

An act which aides the system in determining whether or not to acquireadditional images 1120 can also be implemented. This act may further beaugmented to weigh the suitability of an acquired image. For example, animage detected by a sensor which suffered from excess movement duringthe exposure may be discarded by the algorithm. In this case, the lastacquired image would be discarded and the process would be repeated withthe illumination of the object 1102. In another aspect, the receivedimage with an addressable-pattern may be suitable for processing;however, more images are needed to reconstruct the object. In thisinstance, a further decision process can be added to further augment thealgorithm, an example of which would be to add a decision to determinewhether or not the viewpoint of the imaging system should be adjusted1130. If the position of the imaging device or the desired area of theobject needs to be shifted, either the imaging system or the targetobject may be altered to adjust the viewpoint 1140.

Once all or at least some of the images have been acquired, therelationship amongst points, or point information, within each image isused to calculate or determine the relative or absolute distanceinformation for each point, which is stored as an addressed list. Oncethe distance information is known, the information may be fed to analgorithm which uses the distance information to generate arepresentation of the object 1160.

The drawings and the associated descriptions are provided to illustrateembodiments of the invention and not to limit the scope of theinvention. Reference in the specification to “one embodiment” or “anembodiment” is intended to indicate that a particular feature,structure, or characteristic described in connection with the embodimentis included in at least an embodiment of the invention. The appearancesof the phrase “in one embodiment” or “an embodiment” in various placesin the specification are not necessarily all referring to the sameembodiment.

As used in this disclosure, except where the context requires otherwise,the term “comprise” and variations of the term, such as “comprising,”“comprises” and “comprised” are not intended to exclude other additives,components, integers or steps.

Also, it is noted that the embodiments are disclosed as a process thatis depicted as a flowchart, a flow diagram, a structure diagram, or ablock diagram. Although a flowchart may disclose various steps of theoperations as a sequential process, many of the operations can beperformed in parallel or concurrently. The steps shown are not intendedto be limiting nor are they intended to indicate that each step depictedis essential to the method, but instead are exemplary steps only.

In the foregoing specification, the invention has been described withreference to specific embodiments thereof. It will, however, be evidentthat various modifications and changes may be made thereto withoutdeparting from the broader spirit and scope of the invention. Thespecification and drawing are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It should be appreciatedthat the present invention should not be construed as limited by suchembodiments.

From the foregoing description, it will be apparent that the presentinvention has a number of advantages, some of which have been describedherein, and others of which are inherent in the embodiments of theinvention described or claimed herein. Also, it will be understood thatmodifications can be made to the device, apparatus and method describedherein without departing from the teachings of subject matter describedherein. As such, the invention is not to be limited to the describedembodiments except as required by the appended claims.

1. A device for three-dimensional (3-D) imaging comprising: a lens; atleast one polarization-coded aperture obstructing at least part of thelens; a polarization-sensitive sensor operable for capturingelectromagnetic radiation transmitted from an object through the lensand the at least one polarization-coded aperture; and a processorcommunicatively connected with the sensor for processing the sensorinformation and producing a 3-D image of the object.